Camera controller for aquaculture behavior observation

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for controlling a camera to observe aquaculture feeding behavior. In some implementations, a method includes moving a camera to a first position, obtaining an image captured by the camera at the first position, determining a feeding observation mode, and based on the feeding observation mode and analysis of the image, determining a second position to move the camera.

FIELD

This specification relates to an automated camera controller foraquaculture systems.

BACKGROUND

Aquaculture involves the farming of aquatic organisms, such as fish,crustaceans, or aquatic plants. In aquaculture, and in contrast tocommercial fishing, freshwater and saltwater fish populations arecultivated in controlled environments. For example, the farming of fishcan involve raising fish in tanks, fish ponds, or ocean enclosures.

A camera system controlled by a human operator can be used to monitorfarmed fish as the fish move throughout their enclosure. When camerasystems are manually controlled, human factors, such as the attentionspan or work schedule of the operator, or the comfort of the humanoperator in extreme weather conditions, can affect the quality ofmonitoring.

SUMMARY

In general, innovative aspects of the subject matter described in thisspecification relate to controlling a camera to observe aquaculturefeeding behavior. Farming aquaculture livestock may require that thelivestock be fed while the livestock grows. For example, salmon beingfarmed may be fed for three to seven hours a day until the salmon arelarge enough to be harvested.

Observing feeding behavior may rely on appropriately controlling acamera to observe feeding. For example, if a camera is too far fromfeeding livestock then no feeding behavior may be observed. In anotherexample, if a camera is too close to feeding livestock, then no feedingbehavior may be observed as a single livestock may take up an entireview of the camera. In yet another example, if the camera is too shallowor too deep compared to the depth that the fish are feeding, then nofeeding behavior may be seen. Controlling a camera to observe feedingmay rely on images of the livestock and feed to determine where thecamera should be placed. For example, the camera may be controlled tofind feeding livestock, and then positioned an appropriate distance fromthe feeding livestock to observe feeding behavior.

Feeding behavior of livestock may be observed to obtain usefulinformation. For example, feeding behavior may indicate that livestockare not consuming a large majority of the feed being provided to thelivestock so the amount of feed provided to the livestock may bereduced. In another example, feeding behavior may indicate thatlivestock are quickly consuming feed being provided to the livestock sothe rate that feed is provided to the livestock may be increased. In yetanother example, feeding behavior may indicate that livestock areunhealthy as they are not consuming as much feed as expected somedication may be provided to the livestock.

A system that provides automated control of a camera to observeaquaculture feeding behavior may provide more accurate determination offeeding behavior and may increase efficiency in feeding livestock. Forexample, the automated control may ensure that the camera is optimallypositioned to capture images that show feeding behavior of fish. Inanother example, the automated control may allow a system toautomatically increase a rate that feed is provided to fish while thefish are eating most of the feed, and automatically decrease or stopproviding feed to fish when the fish are not eating most of the feed.Accordingly, the system may decrease an amount of waste of feed used inraising livestock by reducing an amount of unconsumed feed and increaseyield by providing more feed for fish to consume.

One innovative aspect of the subject matter described in thisspecification is embodied in a method that includes moving a camera to afirst position, obtaining an image captured by the camera at the firstposition, determining a feeding observation mode, and based on thefeeding observation mode and analysis of the image, determining a secondposition to move the camera.

Other implementations of this and other aspects include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices. A system ofone or more computers can be so configured by virtue of software,firmware, hardware, or a combination of them installed on the systemthat in operation cause the system to perform the actions. One or morecomputer programs can be so configured by virtue of having instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. For instance,in some aspects determining a feeding observation mode includesdetermining that the feeding observation mode corresponds to a feederlocalization mode, and determining a second position to move the cameraincludes determining from the image that fish are likely feeding in aparticular direction from the first position and determining the secondposition based on the particular direction that the fish are likelyfeeding.

In certain aspects, determining from the image that fish are likelyfeeding in a particular direction from the first position includesdetermining from the image that at a location one or more of a densityof fish satisfies a density criteria, a horizontal swimming speed offish satisfies a speed criteria, a number of fish swimming verticalsatisfies a vertical criteria, a number of mouths of fish openingsatisfies a mouth criteria, or a number of feed satisfies a feedcriteria, and determining the particular direction based on thelocation.

In some implementations, determining that the feeding observation modecorresponds to a feeder localization mode includes determining that afeeding process has started. In some aspects, determining a feedingobservation mode includes determining that the feeding observation modecorresponds to patrol mode, and determining a second position to movethe camera includes determining from the image that feed is sinkingbelow the first position and determining the second position to bedeeper than the first position.

In certain aspects, determining from the image that feed is sinkingbelow the first position includes determining that feed is visible at abottom of the image. In some implementations, determining a feedingobservation mode includes determining that the feeding observation modecorresponds to patrol mode, and determining a second position to movethe camera includes determining from the image that feed is not visibleand determining the second position to be shallower than the firstposition.

In some aspects, determining a feeding observation mode includesdetermining that the feeding observation mode corresponds to patrolmode, and determining a second position to move the camera includesdetermining from the image that feed is visible but not sinking belowthe first position, increasing an amount of feed provided to fish, anddetermining the second position to be deeper than the first position.

In certain aspects, actions include obtaining a second image captured bythe camera at the second position, determining the feed is sinking belowthe second position, and reducing the amount of feed provided to thefish. In some aspects, determining that the feeding observation modecorresponds to patrol mode includes determining that a feederlocalization mode has completed.

The details of one or more implementations are set forth in theaccompanying drawings and the description, below. Other potentialfeatures and advantages of the disclosure will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example feeding behavior monitoring system andan enclosure that contains aquatic livestock.

FIG. 2 is a flow diagram for an example process of controlling a camerato observe aquaculture feeding behavior.

FIG. 3A is a diagram that illustrates a position change of the camerawith a horizontal view.

FIG. 3B is a diagram that illustrates a position change of the camerawith an upwards view.

FIG. 4 is a diagram that illustrates changes in depth of a camera duringfeeding.

FIG. 5 is a diagram that illustrates observation of overfeeding.

Like reference numbers and designations in the various drawings indicatelike elements. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit the implementations described and/or claimed inthis document.

DETAILED DESCRIPTION

FIG. 1 is a diagram of an example feeding behavior monitoring system 100and an enclosure 110 that contains aquatic livestock. A Cartesiancoordinate system is provided for ease of reference. Although FIG. 1shows the enclosure 110 extending in the xy-plane, the enclosure furtherextends in the z-direction, with the positive z-direction extending outof the page of the drawing.

The livestock can be aquatic creatures, such as livestock 120, whichswim freely within the confines of the enclosure 110. In someimplementations, the aquatic livestock 120 stored within the enclosure110 can include finfish or other aquatic lifeforms. The livestock 120can include for example, juvenile fish, koi fish, sharks, salmon, andbass, to name a few examples.

In addition to the aquatic livestock, the enclosure 110 contains water,e.g., seawater, freshwater, or rainwater, although the enclosure cancontain any fluid that is capable of sustaining a habitable environmentfor the aquatic livestock. The feeding behavior monitoring system 100includes a sensor subsystem 102, a sensor position subsystem 104, a feedcontrol subsystem 106, a winch subsystem 108, and a feeder 130.

The feeding behavior monitoring system 100 can be used to monitorfeeding behavior of aquatic livestock. For example, the system 100 maybe used to determine, where, how much, and for long fish are feedingwithin the enclosure 110. Observing feeding behavior may be difficult asthe sensor subsystem 102 may need to be positioned appropriately toobserve feeding behavior. For example, if a sensor subsystem 102 ispositioned too far from where fish are feeding, then no feeding behaviormay be observed. In another example, if a sensor subsystem 102 ispositioned too close to where fish are feeding, then a fish passingimmediately next to the sensor subsystem 102 may block anything elsefrom being sensed besides that fish. In general, a distance of six feetfrom feeding may be an optimal amount of distance to observe feeding.For example, six feet of distance from feeding may allow a camera tohave a view of feed sinking while at the same time having a view ofmultiple fish eating the feed. The optimal distance may vary based onvarious conditions. For example, the optimal distance may be greaterwhen the water is more clear or more sunlight is shining on feed.

The feeding behavior monitoring system 100 may control feeding based onthe feeding behavior that is observed. For example, the system 100 maydetermine that the fish are no longer eating the feed and, in response,stop providing feed. In another example, the system 100 may determinethat the fish are eating the feed but also a large portion of the feedis uneaten by the fish and, in response, reduce a rate that feed isbeing provided to the fish. In yet another example, the system 100 maydetermine that the fish are quickly eating all the feed and, inresponse, increase a rate that feed is being provided to the fish.

The sensor position subsystem 104 can store a current position of thesensor subsystem 102 and generate instructions that correspond to aposition to which the sensor subsystem is to be moved. Additionally, thesensor position subsystem 104 may store one or more of watertemperature, dissolved oxygen, or salinity. In some implementations, thefeeding behavior monitoring system 100 is anchored to a structure suchas a pier, dock, or buoy instead of being confined within the enclosure110. For example, instead of being confined within the enclosure 110,the livestock 120 can be free to roam a body of water, and the feedingbehavior monitoring system 100 can monitor livestock within a certainarea of the body of water.

The sensor position subsystem 104 can generate instructionsautomatically. That is, the sensor position subsystem 104 does notrequire a human evaluation or input to determine the suitability of thecurrent position or the next position of the sensor subsystem 102.

The sensor position subsystem 104 can include one or more computers thatgenerate an instruction corresponding to an x, y, and z-coordinatewithin the enclosure 110. The instruction can also correspond to arotation about an axis of rotation 112 of the feeding behaviormonitoring system 100, the axis of rotation being coextensive with aportion of a cord 114 that extends substantially in the y-direction.Such a rotation changes a horizontal angle of the sensor subsystem 102,the horizontal angle being an angle within the xz-plane at which thesensor subsystem receives sensor input. The instruction can alsocorrespond to a rotation about a pin that connects the sensor subsystem102 to components of the winch subsystem 108. Such a rotation changes avertical angle of the sensor subsystem, the vertical angle beingmeasured with respect to the positive y-axis. The instruction candescribe a possible position, horizontal angle, and vertical angle ofthe sensor subsystem 102 within the enclosure 110.

In some implementations, the sensor position subsystem 104 can becommunicatively coupled to a computer that can present the sensor datato a caretaker of the aquatic livestock who can observe the livestockand the enclosure 110. The sensor position subsystem 104 can communicatethe instruction to the winch subsystem 108.

The sensor position subsystem 104 is communicatively coupled to the feedcontrol subsystem 106. For example, the sensor position subsystem 104may receive information that indicates a rate that feed is beingprovided through the feeder 130. In another example, the sensor positionsubsystem 104 may provide instructions to the feed control subsystem 106to request that the feed control subsystem 106 control the feeder 130 tostart providing feed, stop providing feed, increase a rate that feed isprovided, or decrease a rate that feed is provided. The sensor positionsubsystem 104 may use sensor data to control feeding through the feedcontrol subsystem 106. For example, the feed control subsystem 106 maydirectly control the feeder 130 and the sensor position subsystem 104may determine changes to feeding and instruct the feed control subsystem106 to control the feeder 130 to make those changes.

The sensor position subsystem 104 may position the sensor subsystem 102to observe feeding behavior based on the feeder 130. The feeder 130 maybe one or more of a circular spreader, a linear spreader, or nospreader. A circular spreader may be a rotating spreader that produces acircular distribution of feed, e.g., roughly three to ten meters indiameter (depending on the pressure in the feeding hose and the size ofthe feed). The sensor position subsystem 104 may position the sensorsubsystem 102 so that the winch line is configured to transect thecircle so that there are multiple observation points where the camera isclose to pellet locations.

A linear spreader may be a raised platform that elevates a feeding hoseto spread feed in an ellipse. The sensor position subsystem 104 mayposition the sensor subsystem 102 closer to the center of the ellipsebut generally the position may be less critical given that the feedingzone may be more localized than for a linear spreader.

A no spreader may be similar to the linear spreader without elevation.As a result the feeding zone may be highly localized, e.g., smaller, andthere may be significant crowding of fish, sometimes referred to as avortex, particularly close to the surface. When using a no spreadfeeder, the sensor subsystem 102 may need to be positioned close to thefish to observe feeding behavior.

In some implementations, when dispersion of feed is not that large, feedmay be occluded by fish and the system 100 may rely more on fishbehavior as evidence that feeding is occurring. Accordingly, movingcloser may not generally be that helpful due to extreme occlusion.Additionally or alternatively, the system 100 may overfeed to find thepellets. If the feed rate is increased sufficiently, the pellets willstart to exceed what the fish will eat at the lowest depth that feedingis happening. Either more fish will join lower in the water column orthe pellets will fall though. The system 100 may use the overfeeding toboth find feed as well as determine a maximum allowable feed rate.

The winch subsystem 108 receives the instructions and activates one ormore motors to move the sensor subsystem 102 to the positioncorresponding to the instructions. The winch subsystem 108 can includeone or more motors, one or more power supplies, and one or more pulleysto which the cord 114, which suspends the sensor subsystem 102, isattached. A pulley is a machine used to support movement and directionof a cord, such as cord 114. Although the winch subsystem 108 includes asingle cord 114, any configuration of one or more cords and one or morepulleys that allows the sensor subsystem 102 to move and rotate, asdescribed herein, can be used.

The winch subsystem 108 receives an instruction from the sensor positionsubsystem 104 and activates the one or more motors to move the cord 114.The cord 114, and the attached sensor subsystem 102, can be moved alongthe x, y, and z-directions, to a position corresponding to theinstruction.

A motor of the winch subsystem 108 can be used to rotate the sensorsubsystem 102 to adjust the horizontal angle and the vertical angle ofthe sensor subsystem. A power supply can power the individual componentsof the winch subsystem. The power supply can provide AC and DC power toeach of the components at varying voltage and current levels. In someimplementations, the winch subsystem can include multiple winches ormultiple motors to allow motion in the x, y, and z-directions.

The sensor subsystem 102 can include one or more sensors that canmonitor the livestock. The sensor subsystem 102 may be waterproof andcan withstand the effects of external forces, such as typical oceancurrents, without breaking. The sensor subsystem 102 can include one ormore sensors that acquire sensor data, e.g., images and video footage,thermal imaging, heat signatures, according to the types of sensor ofthe sensor subsystem. For example, the sensor subsystem 102 can includeone or more of the following sensors: a camera, an IR sensor, a UVsensor, a heat sensor, a pressure sensor, a hydrophone, a water currentsensor, or a water quality sensor such as one that detects oxygensaturation or an amount of a dissolved solid.

The feeding behavior monitoring system 100 can additionally store thesensor data captured by the sensor subsystem 102 in a sensor datastorage. In some implementations, the system 100 can store media, suchas video and images, as well as sensor data, such as ultrasound data,thermal data, and pressure data, to name a few examples. Additionally,the sensor data can include GPS information corresponding to ageolocation at which the sensor subsystem captured the sensor data.

One or both of the sensor subsystem 102 and the winch subsystem 108 caninclude inertial measurement devices for tracking motion and determiningposition of the sensor subsystem, such as accelerometers, gyroscopes,and magnetometers. The winch subsystem 108 can also keep track of theamount of cord 114 that has been spooled out and reeled in, to provideanother input for estimating the position of the sensor subsystem 102.In some implementations the winch subsystem 108 can also provide torquesapplied to the cord, to provide input on the position and status of thesensor subsystem 102. In some implementations, the sensor subsystem 102can be attached to an autonomous underwater vehicle (AUV), e.g., atethered AUV.

In the example of FIG. 1, the sensor subsystem 102 includes a camerawhich is fully submerged in the enclosure 110, although in otherembodiments, the sensor subsystem can acquire sensor data withoutcompletely submerging the sensor subsystem, e.g., while the sensorsubsystem is suspended above the water. The position of the sensorsubsystem 102 within the enclosure 110 is determined by instructionsgenerated by the sensor position subsystem 104.

While various examples are given where sensor position subsystem 104determines a position to place the sensor subsystem 102 to observefeeding and determines how to control feeding, other implementations arepossible. For example, instead of the sensor position subsystem 104, thefeed control subsystem 106 may determine how to control feeding. Inanother example, the sensor subsystem 102 may perform the functionalityof both the sensor position subsystem 104 and the feed control subsystem106.

FIG. 2 is a flow diagram for an example process 200 for controlling acamera to observe aquaculture feeding behavior. The example process 200may be performed by various systems, including system 100 of FIG. 1.

The process 200 includes moving a camera to a first position (210). Forexample, the sensor position subsystem 104 may determine that feed isnot visible in an image from the sensor subsystem 102 but a large densegroup of fish is visible, reflecting likely feeding, is visible in thedistance and, in response, transmits an instruction of “move forward” tothe winch subsystem 108.

In another example, the sensor position subsystem 104 may determine thatfeed is visible but a distance from the sensor subsystem 102 to the feedis more than six feet and, in response, transmit an instruction of “moveforward” to the winch subsystem 108. In yet another example, the sensorposition subsystem 104 may determine that many close fish are visibleand no feed is visible, reflecting that the sensor subsystem 102 islikely surrounded by feeding fish, and, in response, transmit aninstruction of “move backwards” to the winch subsystem 108.

The process 200 includes obtaining an image captured by the camera atthe first position (220). For example, the sensor subsystem 102 maycapture images of the livestock 120 feeding on the feed 132. In anotherexample, the sensor subsystem 102 may capture images of the feed 132dropping but no livestock 120 feeding on the feed 132. In yet anotherexample, the sensor subsystem 102 may capture images of the livestock120 feeding on the feed 132 but at a distance more than six feet.

In still yet another example the sensor subsystem 102 may capture imagesthat don't show feed but do show livestock 120 crowded at a location,which indicates that the livestock 120 may be feeding at the location.In a final example, the sensor subsystem 102 may capture images withoutfeed but with many close livestock 120, which indicates that the sensorsubsystem 102 may be too close to feeding.

The process 200 includes determining a feeding observation mode. Forexample, the sensor position subsystem 104 may determine that the system100 is in a feeder localization mode (230). In another example, thesensor position subsystem 104 may determine that the system 100 is in apatrol mode. In some implementations, the feeder localization mode mayoccur before the patrol mode, and the patrol mode may only begin afterthe localization mode is completed. In other implementations, the feederlocalization mode and patrol mode may occur concurrently where theposition of the camera is continually updated based on both modes. Theposition of the camera may evolve both over time (e.g. the feeder movesdue to line changes or drift from current, wind, waves) or with depth(e.g., due to water current pushing the feed as the feed descends).Accordingly, the feed path may be not be vertical but instead a diagonalline from the up current side at the top to the down current side at thebottom) and the camera tracks the feed path using a top camera and/orbottom camera to determine positions for the camera.

The process 200 includes based on the feeding observation mode andanalysis of the image, determining a second position to move the camera(240). For example, the sensor position subsystem 104 may determine tomove the sensor subsystem 102 more forward toward the feed 132.

In some implementations, determining a feeding observation mode includesdetermining that the feeding observation mode corresponds to a feederlocalization mode, and determining a second position to move the cameraincludes (i) determining from the image that fish are likely feeding ina particular direction from the first position and (ii) determining thesecond position based on the particular direction that the fish arelikely feeding. For example, the sensor position subsystem 104 maydetermine that the feeding observation mode corresponds to a feederlocalization, from the image that fish are likely feeding in front of acurrent position where the image was captured, and determine the secondposition to be in front of the current position.

In some implementations, during the feeding observation mode, ensuringfeeding is occurring may be done by getting a current feeding rate. Forexample, the sensor position subsystem 104 may obtain a feed rate fromthe feed control subsystem 106. Ensuring feeding may be complicated bythe fact that there may be a lag between when feed is being released andwhen it is delivered. For example, feed may be stored half a mile fromthe enclosure 110 so may take some time to arrive at the enclosure 110.

Additionally, at some sites feeders may share feeding hoses so feed maybe delivered by the feeder 130 in a duty cycle fashion. For example, thefeeder 130 may provide feed for one minute every three minutes. Feedingperception signals may be clearest when fish are most hungry.Accordingly, the sensor position subsystem 104 may position the sensorsubsystem 102 to observe feeding at the start of feeding, e.g., a fewminutes after feeding has begun.

In some implementations, determining from the image that fish are likelyfeeding in a particular direction from the first position includesdetermining from the image that at a location one or more of a densityof fish satisfies a density criteria, a horizontal swimming speed offish satisfies a speed criteria, a number of fish swimming verticalsatisfies a vertical criteria, a number of mouths of fish openingsatisfies a mouth criteria, or a number of feed satisfies a feedcriteria, and determining the particular direction based on thelocation.

For example, the sensor position subsystem 104 may determine that at alocation in front of the sensor subsystem 102 one or more of: a densitycriteria of fish is more than one fish per cubic foot, a horizontalswimming speed of fish satisfies a speed criteria of an average of tenmiles per hour, a number of fish swimming vertical satisfies a verticalcriteria of two fish per second, a number of mouths of fish openingsatisfies a mouth criteria of three fish per second, or a number of feedsatisfies a feed criteria of three pellets per second, and, in response,determine the particular direction is in front.

In some implementations, determining that the feeding observation modecorresponds to a feeder localization mode includes determining that afeeding process has started. For example, the sensor position subsystem104 may determine that the feed control subsystem 106 has just startedproviding feed and, in response, determine the feeding observation modeis a feeder localization mode.

In some implementations, determining a feeding observation mode includesdetermining that the feeding observation mode corresponds to patrol modeand determining a second position to move the camera includesdetermining from the image that feed is sinking below the first positionand determining the second position to be deeper than the firstposition. For example, the sensor position subsystem 104 may determinethat feed is sinking below a view of the sensor subsystem 102 and, inresponse, determine to position the sensor subsystem 102 four, six,eight feet, or some other distance deeper.

In some implementations, determining from the image that feed is sinkingbelow the first position includes determining that feed is visible atthe bottom of the image. For example, the sensor position subsystem 104may detect feed near a bottom fifth of the image and, in response,determine that feed is sinking below the first position.

In some implementations, determining a feeding observation mode includesdetermining that the feeding observation mode corresponds to patrol modeand determining a second position to move the camera includesdetermining from the image that feed is not visible and determining thesecond position to be shallower than the first position. For example,the sensor position subsystem 104 may determine that feed is not visiblein an image so all the feed may be being consumed above the position ofthe sensor subsystem 102 and, in response, determine the second positionto be eight feet above a current position.

In some implementations, determining a feeding observation mode includesdetermining that the feeding observation mode corresponds to patrol modeand determining a second position to move the camera includesdetermining from the image that feed is visible but not sinking belowthe first position, increasing an amount of feed provided to the fish,and determining the second position to be deeper than the firstposition. For example, the sensor position subsystem 104 may determinethat all feed is being consumed in a current view and that a rate thatfeed is provided may be increased and, in response, increase a rate offeed being provided and reposition the sensor subsystem 102 deeper toobserve whether the feed is sinking and not being consumed.

In some implementations, the process 200 includes obtaining a secondimage captured by the camera at the second position, determining thefeed is sinking below the second position, and reducing the amount offeed provided to the fish. For example, the sensor position subsystem104 may determine from images that feed is sinking below a secondposition so is not being consumed and, in response, instruct the feedcontrol subsystem 106 to reduce a rate that feed is provided to thefish.

In some implementations, determining that the feeding observation modecorresponds to patrol mode includes determining that the feederlocalization mode has completed. For example, the sensor positionsubsystem 104 may enter the patrol mode once the feeder localizationmode has ended when the sensor subsystem 102 has a feeding zone centeredand feed is visible.

FIG. 3A is a diagram that illustrates a position change of the camerawith a horizontal view. As shown, the sensor subsystem 102 includes acamera with a view pointing to the right and the sensor positionsubsystem 104 instructs the winch subsystem 108 to move the sensorsubsystem 102 horizontally towards the livestock 120.

In some implementations, during a feeder localization mode the sensorsubsystem 102 may initially be positioned starting at a shallow depth,e.g., two meters or less, at one extreme side of the enclosure 110 andbe moved horizontally across the enclosure 110 until feeding isdetected. For example, the sensor position subsystem 104 may initiallylook for high fish density as that can be seen from far away andindicate a feeding zone is being approached, then look for an averagehorizontal swimming speed of the fish correlating with feeding, thenlook for vertically swimming fish as those fish may be swimming upwardsto intercept feeding pellets as they fall, then look for fish mouthopening, and then finally look for feed pellets.

Once the feed is detected, the sensor position subsystem 104 may movethe sensor subsystem 102 to maximize an amount of feed seen per second.Additionally, the sensor position subsystem 104 may adjust a pan angleof the sensor subsystem 102 to center feeding in a view of the sensorsubsystem 102. Adjusting a pan angle may occur continuously whilefeeding is being observed.

In some implementations, due to currents, movement of the feeder,changes in blower pressure, or general changes in sea state, the feedingzone may move over time so the sensor position subsystem 104 mayreposition the sensor subsystem 102 from time to time. Repositioning mayfollow the same process as initially positioning the sensor subsystem102, but may also assume a position is generally correct and only usehorizontal and pan movement to at least one of increase an amount offeed seen or center feeding activity.

In some implementations, the system 100 may use a feeder localizationmode that locates a feeding zone based on measuring a distribution offeed detection events as the sensor subsystem 102 moves. The sensorsubsystem 102 may be moved to transect the enclosure 110, e.g., movefrom one side to the other at a depth of three to six meters, to locatethe feeding zone. Because the feeder 130 may deliver feed non-uniformly,e.g., due to clogs, time sharing of feeding hoses, etc., it may benecessary for the sensor position subsystem 104 to move the sensorsubsystem 102 slowly and make multiple passes back and forth.

The sensor position subsystem 104 may move the sensor subsystem 102consistently and measure the exposure time (e.g., in camera frames/unitdistance). For example, if the sensor subsystem 102 takes twice as longin one spot, then the sensor position subsystem 104 may expect to seetwice the number of feed detections at the spot.

As a result of moving the sensor subsystem 102, the system 100 mayobtain a probability distribution that may be used to find a largestpeak, where the location of the largest peak corresponds to an optimalinitial position of the sensor subsystem 102 to observe feedingbehavior. Once in the location that corresponds to the largest peak, thepan angle may be adjusted to maximize the number of feed seen. Forexample, in a two camera system, the pan angle may be adjusted so thatthe amount of feed seen in a side camera may be maximized while a topcamera is used in conjunction to make sure feed is visible.

As the feeding process progresses, both the side stereo camera and thetop camera may be used in conjunction to make sure the camera stays inthe feeding zone. If no feed is seen for some threshold time while thefeeder 130 is running (e.g., based on connection to the feed controlsubsystem 106) then a calibration mode may be started, where thecalibration mode is similar to the feeder localization mode but thecamera is moved to transect for a shorter horizontal distance (e.g.,+/−five meters). If no feed is found by the sensor position subsystem104 during the calibration mode but the feeder 130 is running, then thatmay indicate the feeding hose is blocked and the sensor positionsubsystem 104 may raise an exception that the feeding hose is blocked.

FIG. 3B is a diagram that illustrates a position change of the camerawith an upwards view. As shown, the sensor subsystem 102 includes acamera with a view pointing upwards and the sensor position subsystem104 instructs the winch subsystem 108 to move the sensor subsystem 102to the right towards the livestock 120. The camera with a view pointingupwards may be positioned in an approach similar to the approachdescribed above for the camera with a horizontal view.

FIG. 4 is a diagram that illustrates changes in depth of a camera duringfeeding. In some implementations, the sensor subsystem 102 may be movedto scan vertically through the water column to produce a map over timeof different depths and feeding behavior at those positions. There areseveral strategies for moving the sensor subsystem 102.

One approach may be to scan a full depth, e.g., from six meters to sixtymeters, of the water column each time. However, a disadvantage may bethat the sensor subsystem may need to move slowly to not affect thebehavior of the fish so each scan from top to bottom may take aconsiderable amount of time. Scanning a full depth may take longer toupdate activity as observations may be temporal as well as spatial, sofocusing on critical areas may enable higher frequency of updates.

FIG. 4 illustrates another approach of using feeding behavior, e.g. thepresence/absence of pellets and fish, to control the range that thesensor subsystem 102 travels during feeding. This approach may permithigher frequency scanning of relevant parts of the water column. Forexample, the sensor position subsystem 104 may continuously scan from adepth just before when feeding begins to a depth just after wherefeeding ends.

The graph in the lower left of FIG. 4 shows how the depth of the sensorsubsystem 102 changes across time corresponding to a depth that feed isbeing consumed. The arrows going up and down reflect the depth of thesensor subsystem 102 and the arrow travelling across the up and downarrows reflects depths that feed is finished being consumed. As shown inFIG. 4, the depth of feeding increases as the fish get full and stopfeeding as quickly.

Another approach is using uncertainty in an internal model to targetspecific parts of the water column to collect data, e.g. areas with lowfish density maybe less relevant that areas with high fish density, andthe region at the bottom of the feeding zone may be more critical toobserve than the top of the water column. In still another approach, thesystem 100 may keep a model describing both feeding activity level anduncertainty on feeding. An update model similar to a Kalman filter maybe used to incorporate domain knowledge, such as typical changes in fishbehavior over the feeding period, and observed feeding. This combinedmodel may be tolerant to intermittent degradation in the quality ofobserved feeding, which may be caused, for example, by fish gettingscared of passing boats. An algorithm for sensor subsystem 102positioning may use the combined model so as to reduce uncertainty ofthe current feeding depth.

FIG. 5 is a diagram that shows observation of overfeeding. For example,the sensor subsystem 102 may capture images of the feed sinking below abottom of the enclosure 110 at a sixty foot depth. In someimplementations, the system 100 may determine a maximal feeding rate byobserving overfeeding. For example, the system 100 may determine thatfish aren't being overfeed so may increase a rate that feed is provideduntil the fish are being overfeed.

In more detail, to determine a maximal amount of feed that the fish canconsume at any point in time, the system 100 may feed beyond the maximalamount and then decrease until feed is no longer unconsumed. A maximalfeed acceptance rate may be coupled with a feeding strategy, e.g., e.g.feed rate 90% of a maximal rate. Determining a maximal feed acceptancerate may be done by raising the feed rate by some amount, e.g., 5%, 10%,20%, or some other amount, and positioning the sensor subsystem 102 sixto nine feet below a lowest point where feeding is observed. If the fishconsume all or most of the feed at the new feeding rate, the feedingzone will likely move downwards but all the feed will be observed to beeaten.

Conversely, if the feeding amount is too high, the feed will beunconsumed and may fall through to the bottom of the enclosure 110. Iffeed is not all consumed, then the sensor position subsystem 104 mayslowly decrease a feed rate until feed is no longer being unconsumed.Given there are a large set of fish with complex system dynamics,observations may need to be made over sufficient time, e.g., tens ofminutes, to allow the system 100 to adapt to changes.

The system 100 may similarly be used to detect an end of feeding. Atermination of a feeding process may be when pellets are not beingconsumed and consumption occurring is insufficient to warrant wastingfeeding. For example, the sensor position subsystem 104 may determinethat during a past minute, less than thirty pellets were consumed andhalf of the feed provided was unconsumed and, in response, end thefeeding.

The criteria for terminating feeding may depend on an optimization ofvarious metrics including, but not limited to, a biological feedconversion ratio (e.g., increase in biomass), relative growth index,economic feed conversion ratio (e.g., increase in biomass includingmortalities), environmental factors (e.g., dissolvedoxygen/temperature), and expected appetite based on prior days feeding.The sensor position subsystem 104 may calibrate the criteria based onoptimization of these factors with A/B experimentation.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved.

Embodiments of the invention and all of the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe invention can be implemented as one or more computer programproducts, e.g., one or more modules of computer program instructionsencoded on a computer readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter affecting amachine-readable propagated signal, or a combination of one or more ofthem. The term “data processing apparatus” encompasses all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a tablet computer, a mobile telephone, a personaldigital assistant (PDA), a mobile audio player, a Global PositioningSystem (GPS) receiver, to name just a few. Computer readable mediasuitable for storing computer program instructions and data include allforms of non-volatile memory, media and memory devices, including by wayof example semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, embodiments of the invention canbe implemented on a computer having a display device, e.g., a CRT(cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,e.g., a mouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

Embodiments of the invention can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or thatincludes a middleware component, e.g., an application server, or thatincludes a front end component, e.g., a client computer having agraphical user interface or a Web browser through which a user caninteract with an implementation of the invention, or any combination ofone or more such back end, middleware, or front end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the invention have been described. Otherembodiments are within the scope of the following claims. For example,the steps recited in the claims can be performed in a different orderand still achieve desirable results.

What is claimed is:
 1. A method comprising: moving a camera to a firstposition; obtaining one or more images captured by the camera at thefirst position; determining, from the one or more images, a first valuerepresenting a speed of one or more fish within an enclosure;determining, from the one or more images, a second value representing anumber of open mouths of fish; determining, based on the first valuerepresenting the speed of the one or more fish within the enclosure andthe second value representing the number of open mouths of fish, thatthe one or more fish are likely feeding; and determining a secondposition within the enclosure to move the camera based on determiningthat the one or more fish are likely feeding.
 2. The method of claim 1,wherein determining, based on the first value representing the speed ofthe one or more fish within the enclosure and the second valuerepresenting the number of open mouths of fish, that the one or morefish are likely feeding comprises: determining from the one or moreimages that at a location two or more of: a density of fish satisfies adensity criteria; a horizontal swimming speed of fish satisfies a speedcriteria; a number of fish swimming vertical satisfies a verticalcriteria; a number of open mouths of fish satisfies a mouth criteria; ora number of feed satisfies a feed criteria.
 3. The method of claim 2,wherein determining, based on the first value representing the speed ofthe one or more fish within the enclosure and the second valuerepresenting the number of open mouths of fish, that the one or morefish are likely feeding comprises: determining that an averagehorizontal swimming speeds of the fish shown in the one or more imagessatisfies the speed criteria, wherein the speed criteria correlates withfeeding.
 4. The method of claim 1, comprising: obtaining a second imagecaptured by the camera at the second position; determining, from thesecond image, that a number of fish swimming vertical satisfies avertical criteria; and determining a third position within the enclosureto move the camera.
 5. A computer-readable storage device encoded with acomputer program, the program comprising instructions that when executedby one or more computers cause the one or more computers to performoperations comprising: moving a camera to a first position; obtainingone or more images captured by the camera at the first position;determining, from the one or more images, a first value representing aspeed of one or more fish within an enclosure; determining, from the oneor more images, a second value representing a number of open mouths offish; determining, based on the first value representing the speed ofthe one or more fish within the enclosure and the second valuerepresenting the number of open mouths of fish, that the one or morefish are likely feeding; and determining a second position within theenclosure to move the camera based on determining that the one or morefish are likely feeding.
 6. The device of claim 5, wherein determining,based on the first value representing the speed of the one or more fishwithin the enclosure and the second value representing the number ofopen mouths of fish, that the one or more fish are likely feedingcomprises: determining from the one or more images that at a locationone or more of: a density of fish satisfies a density criteria; ahorizontal swimming speed of fish satisfies a speed criteria; a numberof fish swimming vertical satisfies a vertical criteria; a number ofopen mouths of fish satisfies a mouth criteria; or a number of feedsatisfies a feed criteria.
 7. The device of claim 6, whereindetermining, based on the first value representing the speed of the oneor more fish within the enclosure and the second value representing thenumber of open mouths of fish, that the one or more fish are likelyfeeding comprises: determining that an average horizontal swimmingspeeds of the fish shown in the one or more images satisfies the speedcriteria, wherein the speed criteria correlates with feeding.
 8. Thedevice of claim 5, the operations comprising: obtaining a second imagecaptured by the camera at the second position; determining, from thesecond image, that a number of fish swimming vertical satisfies avertical criteria; and determining a third position within the enclosureto move the camera.
 9. A system comprising: one or more computers andone or more storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform operations comprising: moving a camera to a firstposition; obtaining one or more images captured by the camera at thefirst position; determining, from the one or more images, a first valuerepresenting a speed of one or more fish within an enclosure;determining, from the one or more images, a second value representing anumber of open mouths of fish; determining, based on the first valuerepresenting the speed of the one or more fish within the enclosure andthe second value representing the number of open mouths of fish, thatthe one or more fish are likely feeding; and determining a secondposition within the enclosure to move the camera based on determiningthat the one or more fish are likely feeding.
 10. The system of claim 9,wherein determining, based on the first value representing the speed ofthe one or more fish within the enclosure and the second valuerepresenting the number of open mouths of fish, that the one or morefish are likely feeding comprises: determining from the one or moreimages that at a location two or more of: a density of fish satisfies adensity criteria; a horizontal swimming speed of fish satisfies a speedcriteria; a number of fish swimming vertical satisfies a verticalcriteria; a number of open mouths of fish satisfies a mouth criteria; ora number of feed satisfies a feed criteria.
 11. The system of claim 10,wherein determining, based on the first value representing the speed ofthe one or more fish within the enclosure and the second valuerepresenting the number of open mouths of fish, that the one or morefish are likely feeding comprises: determining that an averagehorizontal swimming speeds of the fish shown in the one or more imagessatisfies the speed criteria, wherein the speed criteria correlates withfeeding.
 12. The system of claim 9, the operations comprising: obtaininga second image captured by the camera at the second position;determining, from the second image, that a number of fish swimmingvertical satisfies a vertical criteria; and determining a third positionwithin the enclosure to move the camera.